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Military  missions  are  more  and  more  focused  upon  peace  enforcing  operations  in  regional  conflicts.  There  is 
often  uncertainty  about  the  intentions,  capabilities  and  strategies  of  the  parties  involved.  Successful 
operations  under  such  unstable  and  complex  conditions  require  competent  commanders  and  staff  personnel. 

Recent  studies  have  shown  that  experts  in  military  tactical  command  treat  decision  making  as  a  problem¬ 
solving  process.  Experts  have  large  collections  of  schemas,  enabling  them  to  recognise  a  large  number  of 
situations  as  familiar.  Another  capacity  of  experts  are  their  problem  solving  skills  if  an  immediate  match 
between  the  actual  problem  situation  and  available  schemas  in  memory  cannot  be  established.  When  faced 
with  an  unfamiliar  tactical  problem,  experts  collect  and  critically  evaluate  the  available  evidence,  seek  for 
consistency,  and  test  assumptions  underlying  an  assessment.  They  then  integrate  results  in  a  comprehensive 
story.  This  expert’s  approach  has  been  used  to  develop  critical  thinking  training. 

This  paper  presents  empirical  studies  into  the  effects  of  critical  thinking  training.  Individual  commanders 
and  commanding  teams  played  scenario-based  exercises  in  both  simplified  and  high-fidelity  task 
environments.  Half  of  the  participants  received  instruction,  guidance,  and  feedback  in  critical  thinking.  The 
other  half  received  the  same  scenarios,  but  without  specific  support.  After  training,  test  scenarios  were 
administered  to  all  groups.  Results  showed  positive  effects  on  the  process  of  tactical  command  (i.e.  better 
argumentation  for  situation  assessment)  as  well  as  on  the  outcomes  (i.e.  more  and  better  contingency  plans). 
In  addition,  members  of  critical  thinking  training  teams  were  more  inclined  to  clarify  their  perspective  of  the 
situation  to  each  other. 

Critical  thinking  training  supports  commanders  in  situation  assessment  and  decision  making,  and  stimulates 
team  members  to  engage  in  activities  required  to  develop  a  shared  mental  model  and  to  co-ordinate  actions. 
For  the  Netherlands  Navy  and  Army,  the  results  of  the  studies  are  now  being  used  to  implement  critical 
thinking  into  tactical  decision  games  in  order  to  develop  new  military  training  programs  that  will  enhance 
the  acquisition  of  sophisticated  domain  knowledge  and  decision  making  skills. 


1.  Introduction 

In  recent  decades  changes  in  the  (inter-)  national  political  situation  has  demanded  a  reflection  on  the  way 
military  personnel  have  to  prepare  for  military  missions.  Peace  keeping  and  peace  enforcing  missions  have 
become  more  varied,  complex  and  unpredictable.  These  missions  have  to  be  fulfilled  with  fewer  personnel, 
in  many  different  circumstances.  At  the  same  time,  the  professional  development  of  our  military  officers  has 
shifted  towards  more  civil  areas  such  as  business  administration,  public  administration,  socio-economic 
studies,  logistics  and  humanities.  But  with  the  number  of  deployments  increasing,  combat  capabilities  and 
skill  in  military  tactics  regain  popularity.  Training  and  education  programs  have  to  be  adapted  to  provide 
military  personnel  and  especially  military  commanders  with  training  aimed  at  the  successful  preparation, 
execution  and  management  of  military  operations  in  unstable  and  complex  conditions  (Lussier,  2003). 


Paper  presented  at  the  RTO  HFM  Symposium  on  “ Advanced  Technologies  for  Military  Training", 
held  in  Genoa,  Italy,  13  -  15  October  2003,  and  published  in  RTO-MP-HFM-101 . 
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Studies  into  expert  strategies  in  tactical  decision  making  have  shown  that  experts  have  large  collections  of 
schemas,  enabling  them  to  recognise  a  large  number  of  situations  as  familiar.  Furthermore,  when  faced  with 
novel  situations,  experts  apply  deliberate  problem  solving  strategies  that  differ  significantly  from  those  of 
novices.  When  faced  with  a  complex  and  unfamiliar  tactical  problem,  experts  collect  and  critically  evaluate 
the  available  evidence,  seek  for  consistency,  and  test  assumptions  underlying  an  assessment.  They  thus  try  to 
integrate  the  results  in  a  comprehensive,  plausible,  and  consistent  story  that  can  explain  the  actual  problem 
situation,  whereas  novices  very  often  consider  aspects  of  the  situation  separately  and  independently.  The 
experts’  approach  is  used  to  develop  a  new  training  concept:  critical  thinking  (CT)  (Cohen,  &  Freeman, 
1997;  Cohen,  Freeman,  &  Thompson,  1997;  Cohen,  Freeman,  &  Thompson,  1998). 

The  critical  thinking  strategy  involves  a  problem  solving  approach  to  new  and  unfamiliar  situations.  It  is  a 
highly  dynamic  and  iterative  strategy,  consisting  of  a  moderately  sized  set  of  methods  to  build,  test  and 
critique  situation  assessments.  These  methods  are  to  some  extent  generalizable  but  they  can  only  be  taught  if 
grounded  in  a  specific  domain  and  trainees  have  already  a  certain  level  of  knowledge  of  that  domain. 
Effective  critical  thinking  training  combines  instruction  with  realistic  practice  (Cohen,  Freeman,  & 
Thompson,  1998).  Practice  in  the  form  of  scenario-based  training  is  considered  an  appropriate  approach  to 
training  competencies  required  in  complex  task  environments  (e.g.  Fowlkes,  Dwyer,  Oser  &  Salas,  1998; 
Oser,  1999).  The  design  of  exercise  scenarios  is  very  important  since  these  have  to  provide  opportunities  to 
practise  critical  thinking  strategies.  Some  guidelines  for  the  design  and  execution  of  scenarios  for  critical 
thinking  training  are  described  in  the  next  paragraph. 

The  effects  of  the  critical  thinking  approach  to  tactical  command  and  control  training  have  been  studied  in 
several  explorative  studies  (e.g.  Cohen  &  Freeman,  1997;  Klein,  McCloskey,  Pliske,  &  Schmitt,  1997). 
These  studies  showed  promising  results  but  they  were  conducted  with  individual  decision  makers,  in 
simplified  training  environments  and  performance  of  critical  thinking  trainees  were  compared  to  trainees 
that  did  not  receive  any  training.  We  conducted  two  training  studies  in  which  we  compared  performance  of 
our  critical  thinking  group  with  the  performance  of  a  control  group  that  conducted  the  exercise  scenarios 
with  standard  instruction  and  feedback.  In  the  first  study  we  focussed  on  the  individual  military  decision 
maker  and  we  conducted  the  training  in  a  very  simplified  training  environment.  In  the  second  training 
experiment  we  studied  teams  in  more  complex  and  dynamic  training  environments.  These  studies  are 
described  in  paragraph  3. 

In  paragraph  4  we  will  briefly  outline  the  approach  to  our  implementation  study  of  critical  thinking  training. 
This  study  is  still  being  undertaken,  so  only  some  preliminary  findings  can  be  reported. 

2.  Scenario  generation  and  instruction  methodology  for  critical  thinking 

training 

In  scenario  based  training,  trainees  prepare,  execute  and  evaluate  exercises  that  are  simplified  situations  of 
the  real  world.  A  scenario  has  a  starting  point  and  depending  on  the  type  of  scenario,  events  are  specified  in 
time  and  space.  Scenarios  can  be  very  structured  in  the  sense  that  all  events  are  scripted,  or  have  a  free  play 
character.  Also,  scenarios  can  differ  in  complexity:  they  may  be  simplified,  leaving  out  many  aspects  of  the 
real  world,  or  they  are  complex,  incorporating  many  aspects  of  the  operational  task  environment. 

Scenario  based  training  provides  trainees  with  the  opportunity  to  build  domain  specific  experience  under 
controlled  and  safe  conditions  (Farmer,  van  Rooij,  Riemersma,  Joma,  &  Moraal,  1999).  By  executing 
training  scenarios,  trainees  may  gain  knowledge  about  typical  problems  and  their  solutions,  thereby 
increasing  their  experience  database  of  situation-response  relationships.  For  critical  thinking  training,  it  is 
important  that  the  training  scenarios  provide  trainees  with  the  opportunity  to  practise  critical  thinking  skills. 
These  involve  (Helsdingen  &  van  den  Bosch,  1999): 

Creating  a  story:  A  story  is  a  comprehensive  assessment  of  the  situation,  in  which  all  the  existing  evidence  is 
incorporated  and  explained  and  assumptions  are  made  about  uncertain  aspects  of  the  situation.  Past,  present 
and  future  are  addressed  in  the  story.  The  purpose  of  story  building  is  to  keep  trainees  from  assessing 
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situations  solely  on  isolated  events.  Instead,  trainees  are  taught  how  they  can  integrate  the  available 
information  into  its  context,  which  may  include  elements  as  the  history  of  events  leading  to  the  current 
situation,  the  presumed  goals  and  capacities  of  the  enemy,  the  opportunities  of  the  enemy,  etc. 


Testing  a  story:  Testing  a  story  is  aimed  at  identifying  incomplete  and  conflicting  information.  They  have  to 
correct  these  problems  by  collecting  more  data,  retrieve  knowledge  from  memory,  or  make  assumptions 
about  the  missing  piece  of  the  story  or  to  resolve  conflicts  in  the  argumentation. 

Evaluating  a  story:  After  a  story  is  constructed,  it  should  be  evaluated  for  its  plausibility.  The  decision  maker 
has  to  take  a  step  back,  identify  critical  or  hidden  assumptions  and  play  the  devils’  advocate  by  falsifying 
these  assumptions,  i.e.  explaining  how  this  assumption  can  be  false  and  building  an  alternative  story. 

Time  management  or  the  Quick  test:  Critical  thinking  is  not  always  appropriate.  Decision  makers  have  to 
evaluate  the  time  available  and  the  consequences  of  their  actions.  In  stressful  situations  such  as  those  often 
encountered  by  military  commanders,  usually  there  is  little  time  to  spare.  The  decision  maker  should  act 
immediately  unless  the  risk  of  a  delay  is  acceptable,  the  cost  of  an  error  is  high,  and  the  situation  is  non¬ 
routine  or  problematic  (Cohen,  Freeman,  &  Thompson,  1998).  Critical  thinking  training  focuses  on  the  way 
trainees  apply  these  criteria. 

Our  guidelines  for  scenario  development  follow  directly  from  these  critical  thinking  skills.  An  exercise 
scenario  can  be  based  on  a  real  world  course  of  events;  however,  some  adjustments  have  to  be  made. 

1 .  The  real  world  course  of  events  is  probably  too  complex,  with  too  many  factors  playing  a  role.  It  is 
important  to  simplify  the  scenario,  especially  for  initial  training  exercises.  Domain  experts  can  identify 
critical  factors  and  some  distracting  evidence. 

2.  The  scenario  should  be  built  in  the  same  way  that  trainees  have  to  build  their  story.  The  developer  and 
domain  expert  have  to  develop  a  coherent  story,  identify  critical  assumptions  and  come  up  with  alternative 
explanations.  This  will  help  to  decide  what  information  has  to  be  presented  to  the  trainee  and  what 
uncertainties  or  conflicting  evidence  have  to  be  introduced  into  the  scenario. 

3.  The  scenario  developer  or  instructor  has  to  consider  the  strategies  by  which  the  trainees  can  correct  their 
stories.  Do  they  have  the  knowledge  to  retrieve  extra  information  from  their  memories;  can  they  collect 
additional  information,  what  kind  of  assumption  could  they  come  up  with  to  resolve  the  gaps  or  conflicts  in 
the  information?  This  consideration  will  help  the  instructor  or  scenario  players  to  anticipate  on  trainees’ 
questions  or  behaviour  during  the  actual  exercise. 

4.  Exercise  scenarios  have  to  be  varied  and  challenging.  This  means  that  the  domain  specific  problems  have 
to  be  challenging  and  varied,  in  order  to  prevent  that  critical  thinking  strategies  become  a  trick. 

5.  What  and  when  trainee  performance  has  to  be  measured  is  something  that  has  to  be  identified  during  the 
scenario  development.  Performance  measures  should  be  aimed  at  the  outcome  and  processes  of  task 
performance.  The  outcome  measures  such  as  situation  reports,  orders,  plans,  and  contingency  plans  are 
domain  specific  and  linked  to  particular  scenario  events.  They  have  to  be  designed  and  evaluated  by  domain 
experts.  The  process  measures  have  a  more  generic  character  and  they  refer  directly  to  the  critical  thinking 
skills.  They  can  only  be  evaluated  by  domain  experts,  within  the  context  of  a  specific  scenario.  Process 
measures  include:  information  processing  (selecting  relevant  information,  story  building,  identification  of 
incomplete  or  conflicting  information),  argumentation  (the  explanations  for  missing  or  conflicting  evidence, 
criticising  assumptions,  coming  up  with  alternative  explanations),  time  management  skills  (make  efficient 
use  of  the  available  time),  and  team  skills  (communication,  supportive  behaviour,  co-ordination,  leadership). 

6.  Test  scenarios  have  to  be  developed  for  the  measurement  of  transfer  of  training.  Performance  on  the  test 
scenarios  has  to  be  evaluated  by  independent  experts,  to  prevent  biased  judgements  by  the  familiar 
instructor.  For  the  interest  of  evaluation  of  training  effectiveness  the  independent  expert  should  also  be  blind 
to  the  experimental  manipulations. 
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3.  Training  studies 


ORGANIZATION 


STUDY  1 

The  first  study  is  conducted  in  the  domain  of  “air  defence”  of  the  Royal  Netherlands  Air  Force,  in  particular 
the  Tactical  Command  Station  (TCS)  of  a  ground-to-air  defence  battalion.  In  an  office  room,  trainee-officers 
played  air-defence  scenarios  under  supervision  of  a  scenario  leader.  The  trainee  played  the  role  of  battle 
captain,  the  scenario  leader  played  all  other  functions  (lower  and  higher  control),  and  introduced  the  scripted 
events  in  the  scenario  (e.g.  battle  damage  reports,  information  about  enemy  movements,  identified  radar 
tracks).  Prior  to  each  training  scenario,  the  trainee  was  provided  with  a  short  description  of  the  political, 
military,  and  civil  background  situation. 

Ambiguous,  incomplete  and  inconsistent  information  was  intentionally  introduced  into  the  scenarios  to 
allow  for  alternative  interpretations  of  events. 


Method 

Design:  A  training-posttest  design  was  used  (see  Table  1).  The  supervising  project  officers  arranged 
participants  according  to  their  tactical  education  and  experience,  and  assigned  matched  pairs  of  trainees 
randomly  to  conditions. 

Briefing  and  instruction:  The  critical  thinking  group  received  a  critical  thinking  tutorial,  followed  by  a 
demonstration  in  which  two  scenario  leaders  (one  of  them  played  the  role  of  trainee)  showed  how  critical 
thinking  should  be  used  in  the  scenarios.  Trainees  of  the  control  group  were  instructed  to  run  the  scenarios 
as  a  normal  command  post  exercise. 

Training  general:  Two  sets  of  three  scenarios  were  used.  Two  scenario  leaders  were  available.  Order  of 
scenario  sets,  and  assignment  of  sets  to  scenario  leaders  was  balanced.  While  performing  the  scenarios, 
trainees  were  asked  to  think  aloud  to  give  the  scenario  leader  access  to  the  assumptions  and  reasoning 
underlying  the  assessments  and  decisions.  At  pre-specified  moments,  the  scenario  leader  “froze”  the 
scenario  for  interventions  (see  below).  After  each  scenario,  the  scenario  leader  filled  in  an  evaluation  form. 

Critical  Thinking  group :  critical-thinking  supporting  schemes  were  available  during 
training.  At  scenario  freezes  and  after  completing  the  scenario,  the  scenario  leader  provided  support  and 
feedback  on  the  critical  thinking  process  (e.g.  by  asking  “which  alternative  explanations  are  possible?”  or 
“how  can  you  verify  that  assumption?”). 

Control  Group :  trainees  received  outcome  feedback  only  (e.g.  “that  was  a  good  decision”, 
or  “you  should  have  issued  that  request  earlier”). 

Test:  Two  test  scenarios  and  two  scenario  leaders  were  available.  Order  of  scenario  and  assignment  of 
scenario  to  scenario  leader  was  balanced.  All  trainees  were  asked  to  think  aloud.  No  support  or  feedback  was 
given. 


Condition 

Instruction 

Training 

Test 

Critical  Thinking- 
group  (N=8) 

Instruction  and 
demos  in  critical 
thinking 

Scenarios  1-6,  with  support  in 

critical  thinking 

process-  and  outcome  feedback 

Scenarios  7-8;  without 
support;  no  feedback 

Control-group 

(N=8) 

No  specific 
instruction 

Scenarios  1-6,  no  support 
outcome  feedback  only 

Scenarios  7-8;  without 
support;  no  feedback 

Table  1:  Research  design 
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ORGANIZATION 


Performance  measures:  outcome  measures  were  used  to  assess  the  quantity  and  quality  of  the  end  result 
(what  is  actually  achieved?);  process  measures  to  describe  the  strategies,  steps  or  procedures  used  to 
accomplish  the  task.  Result  and  contingency  plans  were  used  as  outcome  measures;  information  processing 
and  argumentation  as  process  measures.  Scenario  leaders  evaluated  trainee  performance  on  these  variables 
on  a  10-point  scale.  A  verbal  description  was  used  for  each  scale  point,  ranging  from  “very  poor”  for  score 
1,  to  “excellent”  for  score  10.  Prior  to  the  experiment  proper,  scenario  leaders  had  used  the  results  of  a  pilot- 
study  (using  the  same  scenarios  but  with  different  trainees)  to  come  to  a  common  understanding  of  assigning 
scores. 


Statistical  analysis 

Since  the  performance  measures  are  ordinal  data,  we  had  to  perform  non-parametric  statistical  analysis.  Data 
were  analysed  by  means  of  the  Kruskal-Wallis  test. 

Results 


Figure  1  shows  the  median  scores  on  the  test  scenarios.  The  critical  thinking  group  seemed  to  perform 
slightly  better  than  the  control  group  on  all  variables. 

Significant  differences  between  groups  were  only  found  for  contingency  plans  (Fl(l)=3.91,  p<0.05). 
The  variables  information  processing,  argumentation  and  result  showed  a  similar  pattern,  but  the  differences 
between  groups  were  not  significant  (H(l)=l .62,  p=0.21;  H(l)=2.08,  p=0.15;  and  H(l)=l .23,  p=0.27, 
respectively). 

□  control  group  □critical  thinking  group 


processing  plans* 

*=significant  difference  at  alpha  =  0.05 


/ 


Figure  1:  median  scores  on  the  test  scenarios 


Discussion 

Trainees  of  the  critical  thinking  group  performed  a  little  better  than  the  control  group.  This  indicates  that  the 
critical  thinking  approach  might  be  a  sufficient  tool  for  improving  the  quality  of  tactical  command.  The 
results  of  the  study  had  practical  implications  for  the  military  training  organisation  as  well.  The  scenario 
leaders  discovered  that  this  form  of  training  disclosed  gaps  in  tactical  education  of  the  participating  officers 
that  had  remained  concealed  in  the  large-scale  exercises  constituting  normal  training. 

The  outcomes  corroborate  the  positive  outcomes  of  earlier  explorative  studies  (e.g.  Cohen  &  Freeman,  1997; 
Klein  et  al.,  1997).  Flowever,  further  research  is  needed  to  investigate  a  number  of  questions.  First,  scenario 
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leaders’  scoring  may  have  been  biased  by  their  knowledge  of  the  training  intervention  and  the  design  of  the 
study.  More  independent  assessments  are  needed.  Second,  the  effect  of  training  has  been  studied  in  a 
simplified  task  environment.  Eventually,  critical  thinking  skills  need  to  be  applied  in  the  real  world.  For 
reasons  of  transfer  it  is  necessary  to  investigate  whether  critical  thinking  skills  can  be  successfully  trained  in 
high-fidelity  task  environments.  Finally,  the  training  of  the  first  study  focused  upon  the  individual 
commander,  whereas  tactical  command  is  typically  performed  in  a  team.  The  effects  of  critical  thinking 
training  for  teams  need  to  be  determined.  These  questions  are  addressed  in  study  2. 


STUDY  2 

This  study  investigates  the  effects  of  critical  thinking  applied  to  the  training  of  command  teams  operating  in 
their  natural  task  environments.  It  is  conducted  in  the  domains  of  “anti  air  warfare”  (AAW)  and  “anti  surface 
warfare”  (ASuW)  at  the  Operational  School  of  the  Royal  Netherlands  Navy.  Teams  of  trainees  played  single 
ship  /  single  threat  scenarios  in  a  high-fidelity  tactical  simulator.  The  ASuW  and  AAW  teams  consisted  of 
an  officer  and  petty  officer. 

Method 

Design:  A  training-posttest  design  was  used.  The  supervising  project  officer  arranged  the  eight  participating 
teams  according  to  their  tactical  education  and  operational  experience,  and  assigned  matched  pairs  of  teams 
randomly  to  either  the  “critical  thinking  training”  group  or  the  “control”  group.  The  supervising  project 
officer  selected  two  instructors  for  the  study.  They  were  randomly  assigned  to  conditions. 

Briefing  and  instruction:  Prior  to  the  experiment  proper,  instructors  assigned  to  train  “critical  thinking 
training”  teams  were  extensively  briefed  on  the  critical  thinking  training  method,  as  well  as  how  to  support 
trainees  in  the  application  of  critical  thinking  processes.  Instructors  assigned  to  the  control  team  were  not 
informed  about  the  concept  of  critical  thinking.  They  were  told  to  support  the  teams  as  they  would  normally 
do  in  training.  Instructors  trained  one  team  at  a  time.  The  briefing,  training  and  testing  required  four  days  per 
team. 

The  first  day  of  the  actual  study  was  used  for  briefing  and  instruction  of  the  teams.  The  experimenter  and  the 
assigned  instructor  briefed  the  “critical  thinking”  team  on  the  principles  of  critical  thinking  and  showed  them 
how  to  apply  this  principle  in  paper-based  demonstration  scenarios.  The  control  group  instructor  briefed  his 
team  on  the  itinerary  of  the  coming  days,  and  discussed  a  paper-based  demonstration  scenario  with  them. 

Training  general:  On  the  second  day,  teams  received  two  interactive  role-playing  scenarios  in  a  staff  room 
under  supervision  of  their  instructor.  On  the  third  day,  teams  received  two  scenarios  in  the  tactical  simulator. 
A  scenario  run  took  approximately  two  hours.  See  study  1  for  details  on  how  the  instructor  made 
interventions  to  support  learning. 

Critical  Thinking  group :  The  instructor  encouraged  his  team  to  explicitly  execute  all  critical  thinking 
components  and  he  provided  extensive  guidance  and  feedback  during  and  after  the  scenarios. 

Control  Group'.  The  instructor  supported  the  control  group  teams  as  in  normal  training. 

Test:  On  the  fourth  and  final  day,  teams  were  tested  on  two  test  scenarios  in  the  simulator.  Instructors  were 
not  present.  Two  independent  subject  matter  experts  evaluated  the  performance  of  trainees  individually,  as 
well  as  that  of  the  team.  They  received  the  scenarios  on  paper.  Markers  in  the  scenario  description  prompted 
the  evaluators  to  score  trainee  and  team  performance  at  that  particular  moment,  on  specified  performance 
criteria. 

Evaluators  were  not  informed  about  the  concept  of  training  nor  of  the  purpose  and  the  design  of  the  study. 
Assignment  of  evaluators  to  teams  and  to  scenarios  was  balanced. 

Performance  measures:  the  same  outcome  and  process  measures  as  in  study  1  were  used.  In  addition, 
performance  with  respect  to  time  management  and  team  skills  was  also  scored.  Because  the  evaluators  were 
used  to  using  the  official  NATO  4-point  scale,  it  was  decided  to  use  this  4-point  scale  in  this  study  as  well. 


RTO-MP-HFM-101 :  Paper  10 


Page  6  of  1 0 


ORGANIZATION 


The  verbal  descriptions  for  the  four  scale  points  are,  respectively:  1=  UNsatisfactory,  2=  MArginal,  3= 
SAtisfactory,  and  4=  Excellent. 

Prior  to  the  experiment  proper,  the  experimenter  briefed  the  evaluators  about  the  scoring  procedure  and  how 
to  use  the  scale.  The  results  of  a  pilot-study  were  used  to  arrive  at  a  common  interpretation  of  performance 
measurement. 

Statistical  analysis 

Since  the  performance  measures  are  ordinal  data,  we  had  to  perform  non-parametric  statistical  analysis.  Data 
were  analysed  by  means  of  the  Kruskal-Wallis  test. 

Results 

Data  on  individual  as  well  as  on  team  performance  were  collected  during  training  and  test.  For 
reasons  of  brevity,  performance  data  of  the  two  test  scenarios  will  be  reported  only.  Figure  2  shows  the 
results  on  the  test  scenarios. 
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Figure  2:  median  results  on  the  test  scenarios 

Univariate  tests  showed  significant  differences  for  argumentation  (H(l)=7.5,  p<0.05),  time  management 
(H(l)=11.4,  p<0.05),  contingency  plans  (Hl)=5.6,  p<0.05),  and  team  work  (H(l)=8.7,  p<0.05). 
Performances  on  information  processing  and  actions  were  not  significantly  different  between  groups. 


GENERAF  DISCUSSION 

Critical  thinking  training  produced  positive  effects  on  the  process  of  tactical  command  (i.e.  better 
argumentation  for  situation  assessment)  as  well  as  on  the  outcomes  (i.e.  more  and  better  contingency  plans). 
The  method  supports  not  only  individual  commanders  in  situation  assessment  and  decision  making,  it  is  also 
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particularly  suitable  for  team  members  to  clarify  their  assumptions  and  perspectives  on  the  situation  to  the 
other  team  member(s).  This  is  especially  important  for  developing  shared  mental  models  and  to  co-ordinate 
team  actions  (Stout,  Cannon-Bowers,  &  Salas,  1996).  The  dynamic  and  interactive  nature  of  high-fidelity 
simulator  training  sometimes  provides  too  little  opportunity  for  the  object  of  this  type  of  training:  reflective 
and  critical  task  performance.  This  can  be  overcome  by  preparatory  paper-and-pencil-  and  role-playing 
scenarios,  and  by  introducing  pauses  in  the  simulator-scenarios. 

The  present  and  earlier  studies  (Cohen,  &  Freeman,  1997;  Cohen  et  al.,  1997;  Cohen  et  al.,  1998;  Freeman 
&  Cohen,  1996)  now  provide  sufficient  evidence  warranting  the  implementation  of  this  type  of  training  in 
practical  (military)  training  programs.  Such  implementation  studies  are  needed  to  provide  answers  to 
important  questions,  like:  “how  can  we  integrate  critical  thinking  training  into  an  existing  curriculum?”, 
“what  instruction  and  training  do  observer/trainers  need  for  successful  application  (train  the  trainers)?”, 
“what  is  the  transfer  of  training?”,  and  “what  are  the  long-term  effects?” 


4.  Implementation  of  critical  thinking  training 

Our  training  studies  have  shown  positive  results  for  critical  thinking  training.  Flowever,  these  training 
studies  were  specifically  designed  to  study  the  effects  of  our  training  manipulation.  They  were  mini-training 
courses,  conducted  under  controlled  experimental  conditions.  This  is  very  different  from  any  normal  training 
program.  In  a  regular  training  program,  instructors  usually  have  the  freedom  to  treat  every  student 
differently,  according  to  the  individual  students’  needs.  In  our  training  experiment,  instructors  had  to  treat  all 
students  similar  and  according  to  specific  rules  for  reasons  of  standardisation.  Another  difference  is  the  short 
duration  of  a  training  study  as  compared  to  a  regular  training  program.  A  short  training  intervention  may  not 
provide  the  opportunity  for  trainees  to  have  mastered  critical  thinking  skills. 

The  Operational  school  of  the  Royal  Netherlands  Navy  has  recently  revised  the  training  program  for  CIC 
commanders.  Two  separate  curricula,  one  for  the  air  warfare  commander  and  one  for  the  surface  and 
subsurface  commanders,  are  merged  into  one.  This  demanded  a  review  of  the  classes  and  teaching  materials. 
One  of  the  students’  comments  on  the  training  program  concerned  the  (perceived)  lack  of  correspondence 
between  classroom  instruction  on  the  theoretical  basics  of  naval  warfare  and  the  practical  exercises  in  the 
tactical  trainers.  As  part  of  an  effort  to  introduce  an  integrated  teaching  method  for  theoretical  education  and 
practice,  we  have  started  an  implementation  study  for  scenario-based  critical  thinking  training. 


Approach 

The  implementation  study  involved  many  work  packages,  such  as: 

■  the  investigation  and  evaluation  of  the  current  training  program, 

■  selecting  a  suitable  course, 

■  identification  of  the  appropriate  moments  for  a  training  intervention, 

■  design  of  instruction  materials  for  instructors  and  trainees, 

■  conducting  try-outs  with  domain  experts, 

■  the  actual  implementation  of  the  critical  thinking  module. 

■  evaluation  of  the  training:  qualitative  (interviews)  and  quantitative  (performance  measurement  on 
standard  test  scenarios) 

The  CIC  commander  training  program  consists  of  several  separate  courses.  Our  implementation  study 
focused  on  one  of  these  courses,  i.e.  Anti  Surface  Warfare  (ASuW).  Within  this  course,  trainees  have  to 
attend  classroom  sessions  on  theoretical  background  and  principles  of  ASuW  and  apply  this  knowledge  in 
practical  exercises  in  the  tactical  trainer.  The  scenario-based  critical  thinking  training  aimed  to  facilitate  this 
step  by  introducing  tactical  decision  games  (i.e.  paper-and-pencil  scenarios).  The  scenarios  for  these 
exercises  were  developed  by  experts  of  the  operational  school,  according  to  the  guidelines  set  out  in  this 
paper. 
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Prior  to  the  critical  thinking  training  we  have  instructed  the  scenario  leaders  and  instructors  extensively  on 
the  background  and  principles  of  critical  thinking.  We  developed  readers  for  the  trainees  on  critical  thinking 
training  within  the  context  of  surface  warfare,  and  organised  a  classroom  instruction  on  critical  thinking  for 
trainees.  Observation  protocols  and  performance  measures  were  designed  to  support  instructors  in  their 
tasks. 

Critical  thinking  training  consisted  of  4  sessions  of  4  hours.  The  first  session  was  the  classroom  instruction, 
the  other  three  sessions  were  scenario  based  exercises.  In  each  session  one  scenario  was  discussed.  Trainees 
were  A  scenario  consisted  of  a  description  of  a  situation.  Trainees  were  assigned  a  specific  role  in  this 
scenario  (e.g.  ASuW  commander  or  SAG  commander)  and  had  to  develop  one  or  more  plans.  Trainees  were 
encouraged  explicitly  execute  all  critical  thinking  components.  In  the  first  two  scenarios  a  trainer/scenario 
leader  provided  extensive  guidance  and  feedback  during  and  after  the  scenarios,  in  the  last  scenario  it  was 
expected  that  teams  guided  themselves  through  the  critical  thinking  strategy. 

During  the  exercises,  scenario  leaders  evaluated  the  teams’  critical  thinking  processes  and  their  resulting 
plans.  For  a  description  of  the  performance  measures  see  the  training  studies.  After  the  last  training  session, 
we  asked  trainees  to  fill  out  an  evaluation  form  and  provide  us  with  their  opinion  on  critical  thinking 
training.  Until  now,  the  final  evaluation  has  not  been  undertaken. 


Preliminary  findings 

The  students  were  enthusiastic  and  motivated  to  co-operate.  They  found  that  critical  thinking  helped  them 
reasonably  well  to  systematically  assess  a  situation,  integrate  different  observations  into  a  coherent  story, 
identify  uncertainties  and  justify  assumptions,  and  come  up  with  (contingency)  plans.  They  appreciated  the 
simple  scenario  based  exercises  as  a  suitable  method  for  applying  their  tactical  knowledge  and  practising 
their  skills  in  tactical  decision  making.  Although  some  trainees  did  express  that  they  were  not  completely 
convinced  about  the  surplus  value  of  the  critical  thinking  approach  when  taken  into  account  that  it  takes 
valuable  time.  Our  impression  was  that  some  students  did  not  have  enough  domain  knowledge  to 
incorporate  the  strategy  and  apply  it  as  a  context-dependent  problem  solving  strategy,  instead  they  treated  it 
as  an  independent  and  obligatory  thinking  procedure.  Very  often,  these  students  did  not  even  recognise  the 
uncertainties,  ambiguousness  or  conflicts  in  the  scenario.  Even  during  after  action  reviews  they  labelled 
many  alternative  situation  assessments  as  improbable. 

Critical  thinking  in  teams  is  a  fruitful  strategy  to  develop  a  shared  understanding  of  the  situation,  to  avoid 
misunderstandings,  and  to  develop  contingency  plans  since  criticising  one  another’s  assumptions,  playing 
the  devils  advocate,  and  coming  up  with  alternative  explanations  is  easy. 

The  instruction  and  training  for  the  instructors  and  scenario  leaders  is  a  critical  factor  in  the  implementation 
of  a  training  module  in  a  training  program.  We  found  that  the  instructors  of  the  Operational  School  could  not 
conduct  the  critical  thinking  training  exercises  without  our  constant  support.  This  may  be  due  to  insufficient 
time  to  train  and  prepare  the  instructors. 

We  can  conclude  that  the  critical  thinking  approach  is  a  suitable  strategy  for  scenario  based  training  in 
tactical  command.  The  critical  thinking  approach  encompasses  guidelines  for  the  design  of  effective  training 
scenarios  and  instruction  and  support  for  reflective  decision  making.  It  is  especially  helpful  when  introduced 
in  a  simple  training  environment,  with  sufficient  time  and  support  to  practise  all  components  of  the  strategy. 
There  are  a  few  important  considerations  when  implementing  this  training  strategy.  Instructors  and  scenario 
designers  should  be  extensively  prepared  and  instructed  into  the  principles  and  methods  of  critical  thinking. 
Furthermore,  the  approach  only  works  within  the  framework  of  a  specific  domain  and  trainees  should  have  a 
sufficient  level  of  knowledge  of  that  domain  in  order  to  reflect  critically  on  their  decision  making  process. 

The  long-term  effects  of  critical  thinking  training  are  not  yet  determined.  We  hope  to  be  able  to  conduct  our 
final  evaluations  in  the  near  future,  and  maybe  conduct  a  follow  up  study  to  monitor  task  behaviour  of  our 
trainees  in  their  future  work  environment. 
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